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Abstract 

In this paper, we derive explicit product formulas and positive con- 
volution structures for three continuous classes of Heckman-Opdam hy- 
pergeometric functions of type BC. For specific discrete series of multi- 
plicities these hypergeometric functions occur as the spherical functions 
of non-compact Grassmann manifolds G/K over one of the skew fields 
F = R, C, H. We write the product formula of these spherical functions 
in an explicit form which allows analytic continuation with respect to 
the parameters. In each of the three cases, we obtain a series of hyper- 
group algebras which include the commutative convolution algebras of 
if-biinvariant functions on G as special cases. The characters are given 
by the associated hypergeometric functions. 
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1 Introduction 

There is a well-established theory of hypergeometric functions associated with 
root systems due to Heckman, Opdam and Cherednik which generalizes and 
completes the theory of spherical functions on Riemannian symmetric spaces 
in many respects; see [TT], [S], [H], [TB] as well as the literature cited there. 
In rank one, i.e. for root systems of type BCi, these hypergeometric func- 
tions are known as Jacobi functions and were studied by Flensted- Jensen and 
Koornwinder in a series of papers in the 1970ies. A comprehensive exposition 
is given in [8] . In generalization of the one- variable case, hypergeometric func- 
tions associated with root systems are indexed by continuous parameters (the 
multiplicities) on a given root system. They build up the solutions of the joint 
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eigenvalue problem for an associated system of commuting differential opera- 
tors which generalize the radial parts of all invariant differential operators on a 
Riemannian symmetric space G/ K of the non-compact type. In such geometric 
cases, the root system and multiplicity function are given in terms of the root 
space data of {G,K). In fact, the harmonic analysis associated with such hy- 
pergeometric functions is only the Weyl-group invariant part of a more general 
harmonic analysis associated with a commuting family of differential-reflection 
operators of Dunkl type, the so-called Cherednik operators. The associated inte- 
gral transform, which generalizes the spherical transform on symmetric spaces, 
is studied in detail in There are, in particular, a Paley- Wiener theorem 

and a Plancherel theorem established for this transform. In the geometric cases 
{G,K) is a Gelfand pair, and the corresponding spherical functions satisfy a 
product formula which is intimately connected to the harmonic analysis on the 
commutative algebra of iiT-biinvariant measures on G. In the rank one case, 
a positive product formula and harmonic analysis for Jacobi functions associ- 
ated with general non-negative multiplicities was established by Flensted- Jensen 
and Koornwinder, see [5]. However, apart from theses cases, the existence of 
a positive product formula for multivariable hypergeometric functions and a 
positivity-preserving convolution which would allow for a general L^-theory are 
still open in general. 

A natural idea to extend the convolution from particular geometric cases 
to general multiplicities is analytic continuation of the product formula with 
respect to the multiplicities. There are only three classes of geometric cases with 
an infinite discrete series of multiplicities when the rank is fixed, namely the non- 
compact Grassmann manifolds SOQ{p,q)/SO{p) x SO{q), SU{p,q)/S{U{p) x 
U (g)) and Sp{p, q) / Sp{p) x Sp{q). Their real rank is q and the spherical functions 
are hypergeometric functions of type BC with multiplicities depending on p. 
In the present paper, we carry out the interpolation program in these cases. 
We give an explicit product formula for the spherical functions which allows 
analytic extension with respect to the multiplicity parameter p. This yields 
a product formula for three continuous classes of hypergeometric functions of 
type BC interpolating the group cases. Based on the product formula, we obtain 
a complete picture of harmonic analysis within the framework of commutative 
hypergroups on the associated Weyl chamber. In particular, the hypergeometric 
transform becomes an interpretation as a hypergroup Fourier transform. 

The paper is organized as follows: In Section [21 we calculate the product 
formula for the spherical functions on the Grassmann manifolds. Section [3] 
gives a short account on Heckman-Opdam theory as well as the identification 
of the spherical functions on Grassmann manifolds as hypergeometric functions 
of type BCq. The extension of the product formula to a continuous range of 
multiplicities interpolating the dimension parameter p is carried out in Section 
[31 and Section [51 is devoted to the study of the associated hypergroup algebras 
on the Weyl chamber. A central part of this section is the characterization of 
the bounded multiplicative functions which generalizes well-known results for 
spherical functions. The reasoning here is, however, not based on an integral 
representation but on exponential bounds for the Heckman-Opdam hypergeo- 
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metric functions and their generalized Harish- Chandra expansion. 



2 Spherical functions on Grassmann manifolds 
and their product formula 

We consider the Grassmann manifolds G/K where G is one of the indefinite 
orthogonal, unitary or symplectic groups SOo{p,q), SU{p,q) or Sp(j>,q) with 
maximal compact subgroup K — SO{p) x SO{q), S{U{p) x U{q)) or Sp{p) x 
Sp{q), respectively. For a unified point of view we also consider K as subgroup 
of U{p;¥) X U{q,¥), where U{p;¥) is the unitary group over F = R, C or H. In 
the same way G is a subgroup of the indefinite unitary group U{p,q;¥), which 
is the isometry group for the quadratic form 



on F^^''. To avoid exceptions which will be irrelevant lateron, we shall exclude 
the case p = q and assume that p > q > 1 . 

It is well known that (G, K) is a Gelfand pair (this follows from Corollary 
1.5.4. of [3]). The spherical functions of this pair are characterized as the non- 
zero /C-biinvariant continuous functions ip : G ^ C which satisfy the product 
formula 



where dk denotes the normalized Haar measure of K. This means that the space 
of continuous, if-biinvariant compactly supported functions on G is a commu- 
tative subalgebra of the convolution algebra Gc(G). The space Cc{G//K) on 
the double coset space G//K therefore inherits the structure of a commuta- 
tive topological algebra. The spherical functions of (G, K) provide exactly the 
non-zero continuous characters of this algebra, via f ^ Jq f{x)ip{x)dx. 

To make the product formula explicit, we recall the iilAif -decomposition 
of G. Let and i denote the Lie algebras of G and K. g has the Cartan 
decomposition g = I © p with p consisting of the p + g-block matrices 



Let a be a maximal abelian subalgebra of p. Then G = KAK with A — expa. 
The spherical functions of (G, K) are therefore determined by their values on A. 
Actually, they are already determined by their values on the topological closure 
A+ = exp(a+) if a+ is the positive Weyl chamber associated with an (arbitrary) 
choice of positive roots within the restricted root system A = A(a, g) of g with 
respect to a. We may choose for a the set of all matrices Ht £ Mp+q (F) of the 
form 



|xip + ... + |a:p|2-|j;p+i|2 





(2.1) 
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where t := diag(ti, . . . ,tq) is the q x q diagonal matrix corresponding to t = 
(ti,. . . ,tq) G M'' (here M is considered as a subfield of C and H in the usual 
way). The real rank of G is q, and the restricted root system A = A(a, g) is of 
type BCq with the understanding that zero is allowed as a multiplicity on the 
long roots. In this way the limiting case Bg, which occurs for F = M, is included. 
We identify o with via Ht t, where the coordinates are with respect to 
the standard basis ei, . . . , of R"^. Then the Killing form on a becomes the 
standard Euclidean inner product on R''. Here is a comprehensive table of the 
roots a and their (geometric) multiplicities m(a), that is the dimensions of the 
corresponding root spaces; c.f. Table 9 of [13]. The constant d denotes the 
dimension of F as an R-vectorspace, i.e. d = 1,2,4 for F = R, C, H. 



root a 


multiplicity m(a) = rup^dia) 


a{t) ^±ti - 1 <i <q 


dip - q) 


a{t) = ±2ti ; 1 < i < g 


d-1 


a{t) = ±U ± tj ; 1 < i < j < q 


d 



(2.2) 



Thanks to our restriction p > q, the Weyl group of (a, g) is the hyperocta- 
hedral group in all cases, and as a Weyl chamber we may choose 

0+ := {Ht: iti,...tq) e R with h > t2 > . . . > tq > Q}. 

In our identification of a with R'^, the closed chamber a+ corresponds to the set 

C := e R« : > i2 • ■ • > ^9 > 0}. 

A short calculation gives 

{/ cosht Oqx(p-(7) sinhi \ 
at = 0(p_q)xg Ip-q 0(p~5)xg € Mp+q{¥) : t & C 

\ sinht ^qx(p-q) coshi J 

Consider now ^ 

To obtain t back from g, we write g in p x q block notation as 

/A(5) Big)\ 
^-[Cig) Dig))- 

A short calculation gives 

Dig) ^vcoshtv. (2.3) 

Let speCj,(x) denote the singular spectrum of x e Afg(F), that is 
speCs(x) = ^spec(a;*a;) = (Ai, . . . , A<j) G R'' 
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with the singular values Xi of x ordered by size: Ai > . . . > A, > 0. Equation 
(|2.3p shows that the singular spectrum of D{g) is given by speCg(I?((?)) = 
(coshti, . . . coshtg) —: cosht . Therefore 

t — arcosh(speCs(£'(g)) for each g e KatK, t £ C (2.4) 

where arcosh is also taken componentwise. (Observe that D(g) > Iq and there- 
fore all its singular values are > 1). 

Let us now evaluate the product formula (|2.ip for the spherical functions 
of (G, K) explicitly. As spherical functions are if-biinvariant, it suffices to 
calculate the product formula for arguments g = at,h — Qg & A^. Write 
at G ^+ in p X q-block notation: 



at 



At Bt 
Ct Di 



u 0^ 

Then for at, G ^+ and k — [ ^ ^ ) e X we obtain 



atkas — 

and therefore 



* CtuBs + DtvD, 



D{atkas) — CtuBs + DtvDs — (sinhi|0)w( q~) + cosht w coshs. 



With the block matrix 



do := (^q") e Afp,,(F) 



this can be written as 

D (atkas) — sinh t ctq uctq sinh s + coshi t; coshs. 

Notice that a-Quao £ Mq{¥) is a truncation of u given by the upper left q x q- 
block of a. 

Let (f he a. spherical function of (G, K) and put ip{t) :— cp{at) for t G C. Then 
according to formula (|2.4I) it satisfies 

(p{t)(p{s) — / ^(arcosh(spec^_D(af fcas)))dfc. (2-5) 

In order to achieve a simplification of this formula we first extend the integral 
over K to an integral over U{p;¥) x t/o(<z;F) —: Kq, where Uo{q;¥) denotes 
the connected component of the identity in U{q;¥). If F = H then K = Kq, 
but in the other cases K is a proper normal subgroup of Kq. More precisely, 
letT:={z£F: \z\ ~ 1} and H the group of diagonal matrices H — {dz : 
z e T} C Afp_|_g(F) where the diagonal entries of dz are equal 1 apart from the 
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entry in position {p,p), which is z. Then Kq = Ht<K = TxK. Suppose / is 
a continuous function on Kq of the form 

/(fco) = f{<yQU(7Q,v) for fco = 

Then f{dzk) — f{k) for all z G T and k G K and thus by Weyl's formula, 

/ f{ko)dko = fif f{d,k)dk)dz = / f{k)dk 
Jko Jt J k Jk 

where on each of the involved groups, integration is with respect to the normal- 
ized Haar measure. Thus 

ip{t)(p{s) = / (^(arcosh(spec^(sinht (jQUfJo sinhs + coshi w coshs)))c?udw 

Ju(p,¥) JuoiqS) 

with du and dv the normalized Haar measures on U{p,¥) and Uo{q,¥) respec- 
tively. Here the integrand depends only on v and the truncation (TqUcto, which 
is contained in the closure of the ball 

Bg := {w G Mg{¥) : w*w < I}. 

Under the assumption p > 2q this situation is covered by the following 
reduction lemma, which is a consequence of Corollary 3.3. of Til. Let 

i:=d{q-^) + l 

and for fi £ C with Re fi > ^ — 1, put 

/ A{I -w*w)''~''dw. (2.6) 

Here A{x) denotes the determinant of a; G Mq{¥), which is defined as the 
usual determinant for F = R or C, while for F = H we choose the Dieudonne 
determinant, i.e. A(x) = (detc(x))^/^ when x is considered as a complex matrix 
in the usual way. 

2.1 Lemma. Suppose that p > 2q. Then for continuous f : Bq C, 
/ fia*uao)du = ^— [ f{w)A{I - w*w)P'^^^-'' dw. 

JU{p,¥) Js, 

Proof. Consider the action of the unitary group U{p,¥) on Mp^q{¥) by left 
multiplication, (u, x) ux. The orbit of the matrix (Tq under this action is the 
Stiefel manifold 

Ep,, ^{xe Mp,q{¥) : X*X ^ Iq}. 
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Consider further the map U{p,¥) — >■ Sp,^, u i-^ uctq. The image measure of du 
under this map coincides with the normahzed U {p, F)-invariant measure da on 
^p,q- Therefore 

f{a^uao)du = / f{ala)d(j. 

U(p,V) JSp,, 

But (7q(T is the q x q matrix given by the first q rows of a only. According to 
Corollary 3.3. of [M], 

f{a^cr)da = ^— / /(u;)A(/ - w*w)P'^/^-'-'dw, (2.7) 

which finishes the proof. □ 
We thus obtain 

2.2 Proposition. Suppose that p > 2q. Then the spherical functions ip(t) = 
ip{at) satisfy the product formula 

ip{t)(p{s) — f f ^(arcosh(spec5(sinhiwsinhs+ cosh cosh s)))- 

Jb, Juo{q,¥) 

■ A{I - w*wY'^''^-''dvdw. 

Notice that the dependence on p now occurs only in the density, not in the 
domain of integration. 



3 The spherical functions as 5Cg-hypergeometric 
functions 

In this section, we first provide the necessary background on hypergeometric 
functions associated with root systems. For an introduction to the subject, we 
refer to [H], [12] and part I of ^Sj. In a second part, we identify the spherical 
functions on Grassmann manifolds within this framework. 

Let a be a finite-dimensional Euclidean space with inner product ( . , . ) which 
is extended to a complex bilinear form on the complexification oc of a. We 
identify a with its dual space o* — Hom(a, R) via the given inner product. Let 
i? C o be a (not necessarily reduced) root system and let W be the Weyl group 
of R. For a £ i? we write — 2a/ {a, a) and denote by aa(x) = x — {x, a^)a 
the orthogonal reflection in the hyperplane perpendicular to a. 

A multiplicity function on i? is a function fc : i? — > C which is M^-invariant, 
i.e. k{wa) = k{a) for all a & R. We denote by K the vector space of multiplicity 
functions on R and fix a positive subsystem i?+ of R. For k £ K we put 

Hah- 
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The Cherednik operator in direction ^ G a is the differential-reflection operator 
on oc defined by 

where is the usual directional derivative and e'''(^) := e^"**'^^ for A,^ G Oc- 
For fixed multiplicity k, the operators {T^(fc), ^ G ac} commute. Therefore the 
assignment ^ t-^ ^c(^) uniquely extends to a homomorphism on the symmet- 
ric algebra S{ac) over ac, which my be identified with the algebra of complex 
polynomials on ac- The differential- refiection operator which in this way cor- 
responds to p G 5(ac) will be denoted by T{p,k). Let 5(ac)^ denote the 
subalgebra of VK-invariant elements in S'(ac)- Then for each p G S'(ac)^, the 
Cherednik operator T{p, k) coincides with a W^-invariant differential operator 
on C°°(a)^, the VF-invariant functions from C°°(a). The following theorem es- 
tablishes hypergeometric functions associated with root systems. It was proved 
by Heckman and Opdam in a series of papers, see [51 as well as [H]. 

3.1 Theorem. There exists an open regular set K^'^s c K with {k G K : Rek > 
0} C K^"^^ such that for each k G K^'^^ and each spectral parameter A G ac, the 
hypergeometric system 

T{p,k)f=p{X)f VpG5(ac)'^ (3.1) 

has a unique W -invariant solution f{t) = F{X,k;t) which is analytic on a and 
satisfies /(O) — 1. Moreover, there is a W -invariant tubular neighborhood U 
of a in ac such that F extends to a (single-valued) holomorphic function on 
ac X K^"^^ X U , which is called the hypergeometric function associated with R. 
F{X, fc; t) is W -invariant both in X and t. 

Suppose that k is real. Then for VK-invariant polynomials p with real coef- 
ficients, we have 

Tip, k)F{X,k- .) = p{X)F{X,k-.) 

which shows that 

F{X,k;t) = F(X,k;t) Vi G a. (3.2) 

The uniqueness of the solution to the hypergeometric system also implies the 
equivalence 

F{X, k; .)= F{X', fc; . ) ^ A' G W.X 

Let C^(a)^ denote the W^-invariant functions from C^{a). The hypergeo- 
metric transform of / G C^(a)^ is defined by 

J'fW = f fit)F{~X,k;t)dLu{t) 

J a 

where the measure uj — uj^ on a is given by 

du;{t) = II |e<"'*>/2 - e-<"'*>/Y^"^c^i (3-3) 
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{dt denotes the Lebesgue measure on a). There are Paley- Wiener and Plancherel 
theorems for this transform which are obtained by Weyl-group symmetrization 
of the (non-symmetric) Cherednik transform studied in [TT] ; see also [T2] . Define 
the measure v = i>k on ia by 

where dX denotes the Lebesgue measure on ia and c{. ,k) is the c- function on 
cic, 

-pr r((A,aV) + ifc(f)) ^ r((p(fc),aV) + ifc(f) + fcW) 

' ii r((A, av> + ifc(l) + k{a)) ■ J r((p(fc), av) + ifc(|)) 

(3.4) 

with the convention that k{^) — if j ^ R. 
3.2 Theorem. f/JJ/, Theorems 8.6 and 9.13) 

(1) The hypergeometric transform J- is an isomorphism from C^{a)^ onto 
the W -invariant Paley-Wiener space PW (ac)^ , where PW{ac) consists 
of all holomorphic functions f on ac satisfying the growth condition 

3i?>0, ViVeN: sup(l + |A|)^e-^l^"^l|/(A)| < oo. 

The inverse of T : C~(a)^ Pl^(oc)'^ is given hy 
F-^h{t)^ I h{X)F{\,k;t)dv{X). 

J ia 

(2) Let f.g G C^(a)^ and let a-^- be the Weyl chamber of W corresponding 
to Then 



f{t)g{t)dLo{t) = c / Tf{X)Fg{\)dv{\) 

where c > is a normalization constant. 
According to Propos. 6.1 of [llj . 

|i^(A,fc;t)| < |H/|V2 .g|RcA||t| for t e a,A e ac- 

Thus for / e C^{a)^ and fixed sea, the function A ^ Tf{X)F{\k]s) 
belongs to PW^(ac)^, and we obtain the following 

3.3 Corollary. For sea and f e C^(a)^, the generalized translate 

Tsf{t) := / Tf{\)F{\,k-s)F{\k-t)dv{\) 

again belongs to C^(a)^. Moreover, 

T{tJ){\)= F{\,k;s)Tf{\). 



9 



Let us now turn to the spherical functions on the Grassmann manifolds 
G/K. They are identified with hypergeometric functions of type BCq, as follows: 
Consider a = with the standard inner product ( . , . } and regard the restricted 
root system of G/K as a subset of as described in Section [5] With our 
convention including the case F = R, it is given by 

BCq = {±e„ ±2e„ l<i<q}U {±e, ± e^, 1 < i < j < g} 

where (ei, . . . , e^) denotes the standard basis of K'. The corresponding Weyl 
group W is the hyperoctahedral group, which is generated by permutations and 
sign changes of the ei. Put R := {2a : a G BGq} and R+ :— {2ei, 4ei, 1 < 
i < q} 'J {2(ei ± ej), 1 < i < j < q} and denote the associated hypergeometric 
function by Fbc ■ Let m = mp^d be one of the multiplicity functions on BCq 
in the geometric cases according to table (j2.2p and define k = kp^d on R by 

kp,d{2a) = -mp,d{a), a G BGq. 

Writing k in the form k = (ki, ^2, fca) where ki and fc2 are the values on the 
roots ±2ei and ±4ei, respectively and fcs is the value on the roots 2(±ei ± e^), 
we have 

kpM= {d{p-q)/2,{d-l)/2,d/2). 

The spherical functions oi G/K are then indexed by spectral parameters A G 
and given by 

ip\{at) = ^\[t) = FBc,[i\kp^d\t), t e G. 

This follows from the fact that for k = kp^d, the commutative algebra {D{p,k); p G 
S'(C'^)'^} just represents the radial parts of the algebra of all invariant differen- 
tial operators on G/K, see Remark 2.3. of [J]. 

3.4 Example. The rank one case. Here i?+ = {2,4} C M. We have multiplic- 
ities ki,k2 and p — p{k) = ki + 2^2. According to the example in [11], p.89f, 
the associated hypergeometric function is given by 

FBcA\k-t) = 2j^i(^^, ~^^"^^ ,fci + fc2 + ^;-sinh't). 

With a := ki + k2 — \, P '.— k2 ~ \ and the Jacobi functions ^^"'^"^ as in [8], 
this can be written as 

FBcA^X,k■,t)=^^^'^\t). 

The geometric cases correspond toa = ^ — 1, (3 — | — 1. In Propos. 12. 2( 
the ?7o(l)-integral cancels (use the coordinate transform w := v^^w), and the 
product formula reduces to 

if>{t)ip{s) = I ^(arcoshjcosht coshs -I- w sinh i sinh s | ) • (1 — \w\^)^~^dw 

l^pd/2 Jbi 

= / (^(arcoshjcoshi coshs -|- xi sinht sinhs|)(icr(a;) 
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where = ^\ with q; = ^ — 1, /3=| — 1. The second identity is obtained by 
formula (|2.7p for the sphere Ep i = {a; G : \x\ = 1}. In view of relation (5.24) 
in |5], this formula just coincides with the product formula in rank 1 given in 
Section 7 of g], 

(p-^'^\t)ip"^'^\s) = Ca,i3 I I (y9^"'''''(arcosh|coshtcoshs + re*'^ sinht sinhsj)- 

Jo "'0 

•(1 - r2)"-'3-V2^'+i(sinV')^^rdrdV' (3.5) 

which degenerates for /? = —1/2 (i.e. F = K) to an integral over [—1, 1] with 
respect to (1 — r^)""^/^^?-. 

In fact, formula p.5p was established in jFK) for arbitrary a > f3 > with 
(a,/3) 7^ (^5? ^5)' i-^- arbitrary non-negative root multiplicities different from 
zero. 

4 Continuation of the product formula 

In the following, q and d = dimR(F) are fixed. For G C with Re > 7 — 1 and 
spectral parameter A G C define 

■^^(t) = J^i3C,(»A,fc^;i); iGM« 

with multiplicity 

= (/i-rfg/2,(d-l)/2,d/2). 
If /i = pd/2, then = fcp as in the previous section. 

4.1 Theorem. For /i G C wzi/i i?e/i > 7 — 1, t/ie hyper geometric Junctions Lp^ 
satisfy the product formula 

V'i{t)vl{s) = ('5t*^<5.)(<^^) 
wit/i the probability measures 

{5t*f.Ss){f)^— f I f{d{t,s;v,w))A{I-w*wr-'^dvdw 

Kfj, Jb, JUo(q.V) 

where is given by (j2.6p and the argument is 

d(t, s;v,w) — arcosh(spec^(sinht sinhs + cosh cosh s)). 

This is a partial generalization of formula (13.51) by Flensted- Jensen and 
Koornwinder for BCi to higher rank. 

Proof. The basic idea is analytic continuation with respect to the parameter ^ 
in the right half plane by use of 

Carlson's Theorem (see e.g. ([T], p. 186): Let / be a function which is 
holomorphic in a neighbourhood of {z G C : Re z > 0} satisfying f{z) — O(e'^l^l) 
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for some constant c < vr. Suppose that f{n) = for all n G Nq. Then / is 
identically zero. 

A direct application of Carlson's theorem would require moderate exponential 
growth of the hypergeometric function with respect to the relevant multiplic- 
ity parameter ki in a right half plane. So far however, sufficient exponential 
estimates are available only for real, non-negative multiplicities (Propos. 6.1. 
of [11], and the results of [16!). We therefore proceed in two steps. First, we 
restrict to a discrete set of spectral parameters, for which the hypergeometric 
function is a Jacobi polynomial and the required growth properties are guaran- 
teed. In a second step, we fix a non-negative multiplicity and carry out analytic 
continuation with respect to the spectral parameter, using known bounds on 
the hypergeometric function for non-negative multiplicities. 

To go into detail, let i?^ = {a^ : a £ R} be the root system dual to R, 
= Z.i?^ the coroot lattice and P = {A £ M« : (A,a^> G Z Va G i?} the 
weight lattice of R. Further, denote by P+ = {A G P : (A, a^) > Va G R+} 
the set of dominant weights associated with R^. Then for k G K"^^^ and A G P+, 

Fbc, (A + p{k), fc; t) = c(A -f p{k), k)Px{k; t) 

where c(A, k) is the c- function (|3.4p which is meromorphic on C x K, and the 
Pa are the Heckman-Opdam Jacobi polynomials of type BCq] see [S], equation 
(4.4.10). In our case, p{k) is given by 

q q J q q 

p{k) = (fci -f 2/02)^6, -I- 2/c3^(g-i)e, = {p--^+d-l)J2et +d^(g-i)ei- 

i—1 i—1 i—1 i—1 

Using the asymptotics of the gamma function, one checks that for fixed A G P+, 
the function c(A -|- p(fc^), fc^) is bounded away from zero as /i — > 00 in the right 
half plane 

iJ = {/iGC:Re^>7-l}. 
Indeed, for p — p{k) with k = (fci, fc2, fca) one has 

c(A + p,k) — 

^ A r(A. + pOr(p, + fci) ^ r(^ + ifci)r(f + ifci + fc2) 
l\ r(A, + p, + k,) r{p,) ■ l\ r(A^ + ifci + ^2) r(f + ifcO 

11 -n/Ai+pi-A,-p^ , i\T^fPi-Pi\ 11 



i<j 



Y^h±£i^2^ + k,)T{(^) iir( ^'+^'+"^+^^ +fc3)r(i^) 



As fci — >■ 00 in the half-plane Re ki > 0, the first product is asymptotically equal 
to n^^id) the second one is asymptotically equal to 1, the third product is 
independent of ki, and the last product is asymptotically equal to 1. Thus for 
fixed A, c(A + p, k) is bounded away from zero. 
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According to Proposition I2.2[ the P\{k^; .) with /i = pd/2 [p > 2g) satisfy 
the product formula 

Px[k^-t)Px[k^-s) 

= —[ [ -—^----Pxik,;d{t,s;v,w))A{I~w*wr-''dvdw 

(4.1) 

for aU i, s e M'^. The Jacobi polynomials P\{k; .) have rational coefficients in k 
with respect to the monomial basis e'^, ly £ P. This is shown in Par. 11 of [9], 
but it also follows from the explicit determinantal construction in [T] , Theorem 
5.4. Moreover, as derived in the proof of Theorem 3.6 of [13], the normalized 
integral 

^ ^ \A{I -w*wY-^\dw 



converges exactly if Re /i > 7 — 1 and is of polynomial growth as /i 00 in 
H. Thus for fixed t, s, both sides of (|4.ip are holomorphic in jj, € H and of 
polynomial growth as — )■ 00 in iJ. Moreover, they coincide for all half integer 
values /i = pd/2, p > 2q. Application of Carlson's theorem yields that formula 
(|4.ip holds for all fi £ H. This proves the stated result for spectral parameters 
A + p{k) with A e P+ and k = k^, jji £ H . 

Denote again by C C M'' the closed Weyl chamber associated with R+. In 
order to extend the product formula with respect to the spectral parameter, 
we fix S C as well as k — k^ and restrict to real ^ > 7 — 1 first. Then k 
is nonnegative, and we have the following exponential estimate for Fbc from 
pT| . Proposition 6.1: 

\FBc,[\k-t)\ < |H^|i/2e™'^^-e«.R<=('-'^^*>. 

Let H' := {A G C : ReA G (7°} . Then for X e H' and aU weW, 

Re(wA,<) < Rc(A,i). 

Choose a constant vector a G C° so large that d{t,s;v,w) — a is contained in 
the negative chamber —C for all v e U{q) and all w G Bg. Then consider 

F{X,k:t) e-<^^'^+*>FBc,(A,fc;0. 

The function F is bounded as a function of A G i?'. If the spectral parameter 
is of the form A = A + p{kf^) with A G P+, then by our previous results we have 
the product formula 



F(A,fc^;t)F(A,fc^;s) 
1 



'^M JBg JUo{q,r) 



g{A, d{t,s-v,n,)-a-,-t)pi^^^ k^;d{t, S] V, w))A{I - W* w)"-'' dvdw . 

(4.2) 



13 



Both sides are holomorphic and bounded in A G H' . We are now going to 
carry out analytic extension with respect to A. For this, we choose a set of 
fundamental weights {Ai,...,Ag} C P+ and write A e i?' as A = Y^^^iZiXi 
with coefficients G {z e C : Rez > 0}. Successive holomorphic extension 
with respect to zi, . . . , Zq by use of Carlson's Theorem then yields the validity 
of (|4.2p for all A G H', and thus, by VF-invariance and continuity, for all A G C''. 
This proves the stated product formula for real > 7 — 1 . Analytic continuation 
finally gives it for all /i G ff , which finishes the proof of Thcorcm l4.1l 

□ 

5 Hypergroup algebras associated with Fbc 

The positive product formula of Theorem 14.11 leads to three continuous series 
{d = 1, 2,4) of positivity-preserving convolution algebras on the Weyl chamber 
C which are parametrized by fj,. We shall describe them as commutative hy- 
pergroups, having Hcckman-Opdam hypergeometric functions as characters. In 
the group cases, which correspond to the discrete values n = pd/2, these hyper- 
group algebras are just given by the double coset convolutions associated with 
the Gelfand pairs (G, K) as in Section [2l In the rank one case, they coincide 
with the well-known one-variable Jacobi hypergroups. 

Let us first briefly recall some key notions and facts from hypergroup theory. 
For a detailed treatment, the reader is referred to [7]. Hypergroups generalize 
the convolution algebras of locally compact groups, with the convolution product 
of two point measures Sx and Sy being in general not a point measure again but 
a probability measure depending on x and y. 

5.1 Definition. A hypergroup is a locally compact Hausdorff space X with 
a weakly continuous, associative convolution * on the space Mh{X) of regular 
bounded Borel measures on X , satisfying the following properties: 

1. The convolution product Sx * Sy of two point measures is a compactly sup- 
ported probability measure on X, and supp((52; * Sy) depends continuously 
on X and y with respect to the so-called Michael topology on the space of 
compact subsets of X (see 7]). 

2. There is a neutral element S^ satisfying Se*Sx — Sx — Sx* Se for all x d X. 

3. There is a continuous involution x t-^ x on X such that for all x,y € X, 
e G supp((52; * Sy) is equivalent to x ~ y, and Sx * Sy ^ {Sy * (J^) • Here 
for fi G Affc(X), the measure /i~ is given by f-i.^{A) = /i(A^) for Borel sets 
A ex. 

Due to weak continuity, the convolution of measures on a hypergroup is 
uniquely determined by the convolution of point measures. 

If the convolution is commutative, then {Mh{X), *) becomes a commutative 
Banach-*-algebra with identity Se- Moreover, there exists an (up to a multi- 
plicative factor) unique Haar measure lu, that is a positive Radon measure on 
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X satisfying 

/ f{x * y)duiy) = / f{y)duj{y) for all x € X, f G Ce{X), 
Jx Jx 

where f{x * y) = {S^ * Sy){f). The multipUcative functions of a commutative 
hypergroup X arc given by 

X{X) = {^e C{X) : ^j^O,ip{x*y) = fixMy) Vx, y e X}. 

The decisive object for harmonic anafysis is the dual space of X, defined by 

X := {ip G xi^) '■ V is bounded and ip{x) = ip{x) Va; € X}. 

The elements of X are called characters. As in the case of LCA groups, the 
dual of a commutative hypergroup is a locally compact Hausdorff space with 
the topology of locally uniform convergence. It is naturally identified with the 
symmetric part of the spectrum of the convolution algebra {X, w) . In contrast 
to the group case, X is often a proper subset of xi^)- The Fourier transform 
on L^{X,u>) is defined by f{(p) := J-^ fTpduj. It is injeetive, and there exists 
a unique positive Radon measure n on X, called the Plancherel measure of 
{X,*), such that f f extends to an isometric isomorphism from L^{X,u!) 
onto L'^{X,n). As for groups, there are convolutions between functions from 
various classes of L^'-spaces (or measures) on a hypergroup with Haar measure 
ui. For example, if 1 < p < oo and / G L^{X,uj),g G LP{X,lo), then the 
convolution product 

f*g{x)= / f{x*y)g{y)dw{y) 
Jx 

belongs to Lp{X,uj) and satisfies < ||/||i,u;||fl'l|p,w 

Let us come back to the situation of Section 3. With the notions from there, 
we can now state our main theorem: 

5.2 Theorem. (1) Let > 7 — 1. Then the probability measures given by 

{Ss*^St){f) = — [ [ f{d{s,t;v,w))A{I~w*wr-'^dvdw {5.1) 

''^H Jb^ JUo(q,V) 

for s. f E C define a commutative hypergroup structure Cf^ = (C, *^) on 
the chamber C = q+. The neutral elem,ent is and the involution is the 
identity mapping. The support of Ss *fi Si satisfies 

supp{5s *^ 5t) C {r G C : \\r\\oo < ||s||co + II^IU} 
where || .||cx> is the maximum norm in M'. 



15 



(2) A Haar measure of the hypergroup Cf^ is given by the weight function p.3p 
of the corresponding hypergeometric transform, 

q 

(hj^,{t) = const -J]^ I sinht,p^-''(''-i)-i| cosht,;|''-i • 
1=1 

■ J]^ \cosh{2ti) - cosh{2tj)\'^ dt. 

l<i<j<q 

Proof. (1) It is clear that 5^ *^ 5t is a probability measure on C with 
supp((5s *p St) ~ {d{s,t;v,w) = arcosh(spec^(sinhs sinht + cosh st; cosh i)), 

V e Uo{q,W),w eWq}. 

For the support statement, we denote by \\A\\ the spectral norm of A G F'^'^, 
that is ll^ll = ||speCg(A)||oo (the biggest singular value of A). By the submulti- 
plicativity of || . || we obtain for v and w within the relevant range the estimate 

II sinhs w sinht + cosh s w cosht|| < || sinh s|| || sinhi|| + || cosh s|| || cosht|| 

= sinh||s||oo ■ sinh||t||oo + cosh||s||oo ■ cosh||t||o< 
= cosh(||s||cx3 + ||t||oo)- 

This implies the stated support inclusion. For the weak continuity of the convo- 
lution *^ on Mf,(C), it suffices to verify that for each / G Cb{C), the mapping 
(s,t) f{s t) is continuous. But this is immediate because d{s,t;v,w) de- 
pends continuously on its arguments. To see that *^ is commutative, we note 
that spec^(A) = spec^(A*) for A e F'^^'^, and hence d{t, s; v,w) = d{s,t; v*,w*). 
As the integral in ()5.ip is invariant under the substitution u i— u* = ,w i-^ 
w* , it follows that 6t *^ Sg = Sg *f_i St- For the associativity of *p it suffices to 
verify that 

Sr *M ('^s *p ^t){f) = {Sr *M ^s) *p St{f) 

for all / e C^{W)^ and all r,s,t e C. In view of the Paley- Wiener theorem 
for the hypergeometric transform, both sides are equal to 

( Tf{\)F{\, k- r)F{\, k- s)F{\, fc; t)dv{\). 

This proves the assertion. 

From the explicit form of the convolution it is obvious that is neutral. In 
the discrete cases /i = pd/2 coming from Gelfand pairs, *^ is the convolution 
of a double coset hypergroup. Moreover, supp((5s *^ St) is independent of ^. In 
order to see that the identity mapping is a hypergroup involution for all /i, it 
therefore suffices (by uniqueness of an involution) to show that the zero matrix 
is contained in supp((5t *p St). But 

d{t,t;Iq, —Iq) — arcosh(speCg(— (sinht)^ + (cosht)^)) = arcosh(/g) = 0, 
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which proves the claim. 

(2) Let f,g £ C^{m)^. Notice first that 



f{t)= / Tf{X)FiX,k;t)diy{X) 

by the inversion theorem for the hypergeometric transform (Theorem I3.2[) . As 
F{X,k;s*fj,t) = F{X,k; s)F{X, k;t) for all s, i G C we obtain, with the notation 
of Corollary I3.3[ 

The Plancherel formula (Theorem I3.2p further gives 

/ {Tj)gdLO^ =c / T{Tj)'T^di^ = c f Tf{X)F{X,k;s)y^du{X) 

JC JiC JiC 

Ff{X)F{T,g){X) di^(X) = [ f{T;g)duj^ 

iC JC 



with a constant c > 0. It was used here that F{X, k; s) = F{—X, k; s) = F{X, fc; s) 
for A e iC. Choose now a sequence .g„ G C^(E')^, n € N such that <?„ t 1 
pointwise. Then also Ts(g„) t 1, and the monotonic convergence theorem shows 
that 

{Tsf)dujf, = / fduj^. 

^ JC 

This proves that is a Haar measure of C^. 

□ 

5.3 Lemma. Suppose that ip : C is continuous and multiplicative, i.e. 

ip{s)(p{t) = (p{s *^ t) for all s,t ^ C. 
Then ip = ip'^ with some A e . 

Proof. The proof follows standard arguments. For abbreviation, we write k = 
kf,, cj = and * = In a first step, consider g G C^{m)^. Let p e S'(C«)^ 
be a T4^-invariant polynomial and T{p) — Tij), fc^) the associated Cherednik 
operator. As 

g[s*t)= I Tg[X)F{X,k]s)F{X,k]t)dv{X) 

JiRi 

for all s,t G C, we obtain 
T{p),gis*t)= f Tg{X)p{X)FiX,k;s)FiX,k;t)diy{X) ^T{p)tg{s*t) (5.2) 

JiRi 

and 

T{p),g{s*t)\s^o = T{p)g{t). 
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Suppose now ip is continuous, non-zero and multiplicative on C . Notice first 
that <f(jS) = 1, because is neutral. We extend ip to a. VF-invariant function on 
W and choose g € €"^{^3) with J^cpgduj =1. Recall that the involution of 
the hypergroup is the identity. Thus 



V*9{s)= / ip{s *t)g{t)duj{t) ^ (p{s) 
Jc 

and therefore 

ip{s) = ip*gis) = / Lp{t)Ttg{s)duj{t), 
Jc 

which belongs to C°°(R'') because Ttg ^ C^iW^) for all t according to Lemma 
Further, 



ip{s*t)— j ip{r)Trg{s * t) duj{r) 
Jc 

and therefore 

T{p)sip{s *t) ^ / ip{r)T{p)s{Trg{s *t))duj{r) ^ / ip{r)T{p)t{Trg{s * t)) duj{r) 
Jc Jc 

= T{p)t^{s*t). 

In particular, 

T{p)'p{t) = T{p)tip{s *t)\s=o = T{p)s(p{s *t)\s=Q =a^{p)-(p{t) 

with <J^(p) = {T{p)(p){0). The mapping p t-^ o;^{p) is obviously multiplicative 
and linear on S{C)^ . According to a well-known result form invariant theory 
(see e.g. [6], Ch. III. 4, Lemma 3.11), it coincides with a point evaluation, that 
is 

3 A e C : = p{X) \fp £ 5(C'')^. 

It is thus shown that ip satisfies the hypergeometric system (j3.ip withspectral 
parameter A, corresponding to i? = BCq and k — kp^. By uniqueness of the 

-iX- 

□ 



solution, it follows that ip — FBCqi^ikfi] .) = p't 



5.4 Theorem. The set of multiplicative functions and the dual space of the 
hypergroup are given by 

x{C,)^{px = pl: XeC + iC}- 

(Tp^ipxe xiCp) : A e M^.A and Im A G co(W.p)} 

where p = p{k^^) and co{W.p) denotes the convex hull of the Weyl group orbit 
W.p. 

The second part of this theorem is in accordance with the characterization 
of the bounded spherical functions of a Riemannian symmetric space of non- 
compact type, see [5], Chap. IV, Theorem 8.1. In our more general context, we 
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shall not work with an integral representation but proceed by using estimates 
on the hypergeometric function given in |16j as well as the generalized Harish- 
Chandra expansion of [TT]. We mention at this point that for the Grassmann 
manifolds over F = R, there is an explicit integral formula for the spherical func- 
tions given in |15j which could probably also be used after analytic extension. 

Proof. (Proof of Theorem 15.41 ) The identification of x(C'^) is furnished by the 
previous lemma. For the identification of the dual space, note first that 

as a consequence of p.2p . Thus (p\ is real if and only if A G W.\. It remains 
to identify those functions from x(C'a') "which are bounded. For this, we observe 
first that the set A = {X e : ipx £ C^} is closed in C. Indeed, suppose 
that (Ai)igN is a sequence in A which converges to Aq G C''. Being members of 
a hypergroup dual, the ip\. are uniformly bounded by 1 (see [7]). As (A,i) i— >■ 
ip\{t) is continuous, it follows by a standard compactness argument that the 
sequence (p\. converges to ipxg locally uniformly on C (see e.g. [2], Chap. XII, 
Sec. 8). This implies that (p\g belongs to as well. 
Next recall that 

and notice that = F\. We thus have to prove that F\ is bounded if and 
only if Re A e co(W.p). We may assume that X = S^+iij with ^, 77 G C. By Cor. 
3.1 of ^6j, 

\Fx{t)\ < F^t) Vt G C. (5.3) 

Further, according to Remark 3.1 of [loc.cit], F^ behaves asymptotically (for 
large arguments in C) as 

Fdt)^e(^-P-'> ■ n + (5-4) 

Here Rq are the indivisible positive roots, in our case Rq = {2ei, 2(ei±ej), 1 < 
i < 3 ^ q}- Consider now A = ■^+^'7 with C = R-e A G co(W./9). We claim that Fx 
is bounded. By closedness of A, it suffices to assume that ^ is actually contained 
in the open interior of co(W.p). Then there exists a constant < s < 1, s = 1 — e, 
such that ^ G co(W.sp). We use the characterization 

co{w.x) = Pi w{x — C*) (5-5) 

for a: G C, where C* = {x e W : {t, x) > Vi G C} is the closed dual cone of 
C; see e.g. [6], Lemma IV. 8. 3. This shows that sp — ^ G C* and therefore 

(e - P, i) - (e - sp, t) - t) < -e{p, t) yteC. 
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Note that (p, t) > for all t G C \ {0}, because our multiplicity is non- negative 
and different from zero. Hence {p,t) > c\t\ for some constant c > 0. Together 
with estimates (|5.3p and (|5.4p . this proves boundedness of F\ as claimed. 

For the converse inclusion, we have to show that F\ is unbounded if ^ = RcA ^ 
co(W.p). For real X — ^ £ C we use again (|5.4p . According to (|5.5p . there exists 
some < e C such that - t) > (recaU that {^,t) > {w£,,t) for all w G VF). 
This implies that is unbounded in C. 

In case 77 = ImA ^ we employ the Harish-Chandra expansion of Fx (see [TT] ) 
in the interior C° of C. It is of the form 

with (unique) coefficients rq{wX) G C, where ro(wA) — 1. Here is the 
positive lattice generated by and c(A) = c(A, fc^) denotes the c-function. 

As 1^ € C \ co(W.p), there is some t £ C and hence also some to £ C° such 
that — p, ^o) > 0. Fix and consider F\{stQ) for s G M, s +00. As the 
imaginary part of A is nonzero, Lemma 4.2.2. in Part I of [S] implies that there 
exist constants M^x > (depending on io) such that 

|r,(u'A)| < M^Ae^'''*"^ for all q £ Q+ . 
For s G M, s > we may therefore estimate 

I J2 r,(u;A)e' I <M^A ^ e(i-")<«'*«> , 

qeQ+\{0} qeQ+\{0} 

which tends to zero as s +00. Thus 

Fxisto) X J2 <w\)e^'"'^'P-''°^ = J2 c(wA)e''<"'«^'''*«>e'^<""''*°> as s +00. 

Notice that c(A) ^ 0. Moreover, — p,to) > {w^ — p,to) for all w £ W where 
equality can only occur if wS, — Therefore, the leading term of the last sum is 

es(?-P,to) . c(w;A)e*^<""''*'') 

with = {w £ W : — Application of Lemma 15.51 below now implies 
that s t— F\(sto) is unbounded as s — ^ +00. This finishes the proof. □ 

5.5 Lemma. Let f{s) = e"* • "^fe 6*'^'='' on R with constants a > 0, 

Ck £ C which are not all zero, and distinct A/j G M. Then f is unbounded on 
[0,^). 

Proof. Let T > 0. Then according to Corollary 2 of [TU] . 

fe=i 
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with a constant S > depending on the and |6'| < 1. If / were bounded on 
[0,00), say I/I < M, this would imply that 

Jo 'fi Jo 2a 

a contradiction. □ 



Notice that only the first part of our proof of I5.4l uses uniform boundedness 
of hypergroup characters in order to settle boundedness of -Fa in the case where 
Re A is contained in the boundary of co{W.p). The rest of the proof works 
equally for arbitrary root systems R and arbitrary non-negative multiplicities 
fc > 0, fc ^ 0, and the case = is classical. Actually, we have 

5.6 Corollary. Let R G a be an arbitrary root system, k > a non-negative 
multiplicity function and p = p{k). Then the associated hyper geometric function 
1 1~> F{X, fc; t) is unbounded on a if Re X ^ co{W.p). Moreover, 1 1— )■ F{\, k; t) is 
bounded on a if Re X is contained in the interior of co{W.p). 

We return to our specific _BC-cases and identify the dual space of the 
hypergroup with a subset of via (p\ A. Due to the condition A G 
{w.X, w e W} it is contained in the union of finitely many hyperplanes in 
C = R^^ of real dimension q. Notice that the chamber C is a proper subset of 

The following is an immediate consequence of Opdam's Plancherel theorem 
(Thm. 1X211 : 



5.7 Proposition. The Plancherel measure of the hypergroup Cf^ is given by the 
measure 

1 

|c(iA, fc^) 

on C C. Rs support coincides with the chamber C . 
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